Imaging device and electronic device

ABSTRACT

The present technology relates to an imaging device and an electronic device that enable construction of an imaging device that outputs information required by a user with a small size. A single-chip imaging device includes: an imaging unit in which a plurality of pixels is arranged two-dimensionally and that captures an image; a signal processing unit that performs signal processing using a captured image output from the imaging unit; an output I/F that outputs a signal processing result of the signal processing and the captured image to an outside; and an output control unit that performs output control of selectively outputting the signal processing result of the signal processing and the captured image from the output I/F to the outside. The present technology can be applied to, for example, an imaging device that captures an image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/986,049, filed Aug. 5, 2020, which is a continuation application ofU.S. application Ser. No. 16/330,468 filed Mar. 5, 2019, now U.S. Pat.No. 10,795,024, which is a national stage application under 35 U.S.C.371 and claims the benefit of PCT Application No. PCT/JP2017/031539having an international filing date of Sep. 1, 2017, which designatedthe United States, which PCT application claimed the benefit of JapanesePatent Application No. 2016-181194 filed Sep. 16, 2016, the entiredisclosures of each of which are incorporated herein by reference.

TECHNICAL FIELD

The present technology relates to an imaging device and an electronicdevice, and particularly to an imaging device and an electronic devicethat enable construction of an imaging device that outputs informationrequired by a user with a small size, for example.

BACKGROUND ART

An imaging device in which chips such as a sensor chip, a memory chip,and a digital signal processor (DSP) chip are connected in parallelusing a plurality of bumps has been proposed as an imaging device thatcaptures an image (for example, see Patent Document 1).

CITATION LIST Patent Document

Patent Document 1: Japanese Patent Application Laid-Open No. 2008-048313

SUMMARY OF THE INVENTION Problems To Be Solved By The Invention

In a case where chips are connected using bumps to construct an imagingdevice as in the imaging device described in Patent Document 1, thethickness of the imaging device increases and the size of the imagingdevice increases.

On the other hand, a user who uses an imaging device may require animage which is captured by the imaging device and may also requireinformation (meta data) which is acquired from the image, not the imageitself.

The present technology is made in consideration of the above-mentionedcircumstances and an objective thereof is to enable construction of animaging device that outputs information required by a user with a smallsize.

Solutions To Problems

An imaging device or an electronic device according to the presenttechnology is a single-chip imaging device including: an imaging unit inwhich a plurality of pixels is arranged two-dimensionally and thatcaptures an image; a signal processing unit that performs signalprocessing using a captured image output from the imaging unit; anoutput I/F that outputs a signal processing result of the signalprocessing and the captured image to the outside; and an output controlunit that performs output control of selectively outputting the signalprocessing result of the signal processing and the captured image fromthe output I/F to the outside, or an electronic device including theimaging device.

In the imaging device and the electronic device according to the presenttechnology, signal processing using a captured image which is outputfrom the imaging unit in which a plurality of pixels is arrangedtwo-dimensionally to capture an image is performed, and the signalprocessing result of the signal processing and the captured image areselectively output to the outside from the output I/F that outputs thesignal processing result of the signal processing and the captured imageto the outside.

The imaging device may be an independent device or may be an internalblock constituting an independent device.

Effects of The Invention

According to the present technology, it is possible to enableconstruction of an imaging device that outputs information required by auser with a small size.

Incidentally, the effects described herein are not restrictive and anyeffect described in the present disclosure may be achieved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofan embodiment of a digital camera according to the present technology.

FIG. 2 is a block diagram illustrating an example of a configuration ofan imaging device 2.

FIG. 3 is a perspective view illustrating an example of an outlineconfiguration of the imaging device 2.

FIG. 4 is a diagram describing an outline of a processing example of theimaging device 2 in a case where a recognition process is performed assignal processing of a DSP 32.

FIG. 5 is a diagram describing an outline of a processing example of theimaging device 2 in a case where a recognition process is performed assignal processing of the DSP 32.

FIG. 6 is a diagram describing an outline of a processing example of theimaging device 2 in a case where a recognition process is performed assignal processing of the DSP 32.

FIG. 7 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

FIG. 8 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

FIG. 9 is a timing diagram describing a third example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

FIG. 10 is a timing diagram describing a fourth example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

FIG. 11 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where a fusion process is performed assignal processing of the DSP 32.

FIG. 12 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where a fusion process is performed assignal processing of the DSP 32.

FIG. 13 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where a fusion process is performed assignal processing of the DSP 32.

FIG. 14 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where a fusion process isperformed as signal processing of the DSP 32.

FIG. 15 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where a fusion process isperformed as signal processing of the DSP 32.

FIG. 16 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where an SLAM process is performed assignal processing of the DSP 32.

FIG. 17 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where an SLAM process is performed assignal processing of the DSP 32.

FIG. 18 is a diagram describing an outline of a processing example ofthe imaging device 2 in a case where an SLAM process is performed assignal processing of the DSP 32.

FIG. 19 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

FIG. 20 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

FIG. 21 is a timing diagram describing a third example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

FIG. 22 is a block diagram illustrating another example of theconfiguration of the imaging device 2.

FIG. 23 is a diagram illustrating an example of use of the imagingdevice 2.

FIG. 24 is a block diagram schematically illustrating an example of aconfiguration of a vehicle control system.

FIG. 25 is a diagram illustrating an example of an installation positionof an imaging unit.

MODE FOR CARRYING OUT THE INVENTION Embodiment of Digital CameraAccording to Present Technology

FIG. 1 is a block diagram illustrating an example of a configuration ofan embodiment of a digital camera according to the present technology.

Incidentally, a digital camera can capture any of a still image and amoving image.

In FIG. 1, the digital camera includes an optical system 1, an imagingdevice 2, a memory 3, a signal processing unit 4, an output unit 5, anda control unit 6.

The optical system 1 includes, for example, a zoom lens, a focusinglens, an aperture stop, and the like, which are not illustrated andallows external light to be incident on the imaging device 2.

The imaging device 2 is, for example, a complementary metal oxidesemiconductor (CMOS) image sensor constituted by a single chip andserves to receive incident light from the optical system 1, to performphotoelectric conversion, and to output image data corresponding to theincident light from the optical system 1.

Further, the imaging device 2 performs, for example, a recognitionprocess of recognizing a predetermined recognition object and othersignal processing using the image data or the like and outputs a signalprocessing result of the signal processing.

The memory 3 temporarily stores the image data or the like output fromthe imaging device 2.

The signal processing unit 4 performs processes such as noise removaland adjustment of white balance as camera signal processing using theimage data stored in the memory 3 if necessary and outputs the processresults to the output unit 5.

The output unit 5 outputs the image data from the signal processing unit4 or the signal processing result stored in the memory 3.

That is, the output unit 5 includes, for example, a display (notillustrated) including liquid crystal or the like and displays an imagecorresponding to the image data from the signal processing unit 4 as aso-called through image.

Further, the output unit 5 includes, for example, a driver (notillustrated) that drives a recording medium such as a semiconductormemory, a magnetic disk, or an optical disc and records the image datafrom the signal processing unit 4 or the signal processing result storedin the memory 3 on the recording medium.

Furthermore, the output unit 5 functions, for example, as an interface(I/F) that performs data transfer to and from an external device andtransmits the image data from the signal processing unit 4, the imagedata recorded on the recording medium, or the like to the externaldevice.

The control unit 6 controls constituent blocks of the digital camera inresponse to a user's operation or the like.

In the digital camera having the above-mentioned configuration, theimaging device 2 captures an image. That is, the imaging device 2receives incident light from the optical system 1, performsphotoelectric conversion, acquires image data corresponding to theincident light, and outputs the image data.

The image data output from the imaging device 2 is supplied to thememory 3 and stored therein. The image data stored in the memory 3 issubjected to camera signal processing by the signal processing unit 4,and image data acquired as a result is supplied to the output unit 5 andoutput therefrom.

Further, the imaging device 2 performs signal processing using an image(data) and the like acquired by capturing an image and outputs a signalprocessing result of the signal processing. The signal processing resultoutput from the imaging device 2 is stored, for example, in the memory3.

In the imaging device 2, outputting of an image acquired by capturing animage and outputting a signal processing result of signal processingusing the image or the like are selectively performed.

Example of Configuration of Imaging Device 2

FIG. 2 is a block diagram illustrating an example of a configuration ofthe imaging device 2 illustrated in FIG. 1.

In FIG. 2, the imaging device 2 includes an imaging block 20 and asignal processing block 30. The imaging block 20 and the signalprocessing block 30 are electrically connected to each other viaconnection lines (internal buses) CL1, CL2, and CL3.

The imaging block 20 includes an imaging unit 21, an imaging processingunit 22, an output control unit 23, an output interface (I/F) 24, and animaging control unit 25 and captures an image.

The imaging unit 21 has a configuration in which a plurality of pixelsis arranged two-dimensionally. The imaging unit 21 is driven by theimaging processing unit 22 and captures an image.

That is, light from the optical system 1 (FIG. 1) is incident on theimaging unit 21. Each pixel of the imaging unit 21 receives incidentlight from the optical system 1, performs photoelectric conversion, andoutputs an analog image signal corresponding to the incident light.

Incidentally, the size of an image (signal) output from the imaging unit21 can be selected, for example, from a plurality of sizes such as 12M(3968×2976) pixels or a video graphics array (VGA) size (640×480pixels).

Further, an image selected from the imaging unit 21 can be set to beselected, for example, from a color image of RGB (red, green, and blue)and a monochrome image with only luminance.

This selection can be carried out as a kind of setting of an imagingmode.

The imaging processing unit 22 performs an imaging process associatedwith capturing an image in the imaging unit 21 such as driving of theimaging unit 21, analog-to-digital (AD) conversion of an analog imagesignal output from the imaging unit 21, and imaging signal processing,according to the control of the imaging control unit 25.

Here, examples of the imaging signal processing include a process ofcalculating brightness for each predetermined sub area, for example, bycalculating an average value of pixel values for each sub area in animage output from the imaging unit 21, a process of converting an imageoutput from the imaging unit 21 into a high dynamic range (HDR) image,defect correction, development, and the like.

The imaging processing unit 22 outputs a digital image signal (herein,for example, an image with 12M pixels or a VGA size), which is acquiredby AD conversion or the like of an analog image signal output from theimaging unit 21, as a captured image.

The captured image output from the imaging processing unit 22 issupplied to the output control unit 23 and is supplied to an imagecompressing unit 35 of the signal processing block 30 via the connectionline CL2.

The output control unit 23 is supplied with a captured image from theimaging processing unit 22 and is also supplied with a signal processingresult of signal processing using the captured image or the like fromthe signal processing block 30 via the connection line CL3.

The output control unit 23 performs output control of selectivelyoutputting the captured image from the imaging processing unit 22 andthe signal processing result from the signal processing block 30 fromthe (single) output I/F 24 to the outside (for example, the memory 3 orthe like in FIG. 1).

That is, the output control unit 23 selects the captured image from theimaging processing unit 22 or the signal processing result from thesignal processing block 30 and supplies the selected one to the outputI/F 24.

The output I/F 24 is an I/F that outputs the captured image and thesignal processing result supplied from the output control unit 23 to theoutside. For example, a relatively fast parallel I/F such as a mobileindustry processor interface (MIPI) or the like can be employed as theoutput I/F 24.

The output I/F 24 outputs the captured image from the imaging processingunit 22 or the signal processing result from the signal processing block30 to the outside in accordance with the output control of the outputcontrol unit 23. Accordingly, for example, in a case where the outsiderequires only the signal processing result from the signal processingblock 30 and does not require the captured image, only the signalprocessing result can be output and it is thus possible to reduce anamount of data which is output from the output I/F 24 to the outside.

Further, by causing the signal processing block 30 to perform signalprocessing through which a signal processing result required from theoutside and to output the signal processing result from the output I/F24, signal processing does not need to be performed in the outside andit is thus possible to reduce a load of an external block.

The imaging control unit 25 includes a communication I/F 26 and aregister group 27.

The communication I/F 26 is a first communication I/F such as a serialcommunication I/F such as an inter-integrated circuit (I2C) and performsexchange of necessary information such as information which is read fromand written to the register group 27 with the outside (for example, thecontrol unit 6 or the like in FIG. 1).

The register group 27 includes a plurality of registers and stores avariety of information such as imaging information associated withcapturing an image in the imaging unit 21.

For example, the register group 27 stores imaging information receivedfrom the outside by the communication I/F 26 or results of imagingsignal processing (for example, brightness for each sub area in acaptured image or the like) in the imaging processing unit 22.

Examples of the imaging information stored in the register group 27include (information indicating) ISO sensitivity (an analog gain at thetime of AD conversion in the imaging processing unit 22), an exposuretime (a shutter speed), a frame rate, a focus, an imaging mode, acutting area, and the like.

The imaging mode includes, for example, a manual mode in which anexposure time, a frame rate, and the like are manually set and anautomatic mode in which they are automatically set depending on scenes.The automatic mode includes, for example, modes according to variousimaged scenes such as a night scene and a face of a person.

Further, the cutting area refers to an area which is cut from an imageoutput from the imaging unit 21 in a case where the imaging processingunit 22 cuts a part of the image output from the imaging unit 21 andoutputs the cut image as a captured image. By designating the cuttingarea, only an area in which a person appears can be cut, for example,from an image output from the imaging unit 21. Incidentally, cutting ofan image can employ a method of cutting an area from an image outputfrom the imaging unit 21 and a method of reading only an image (signal)in a cutting area from the imaging unit 21.

The imaging control unit 25 controls the imaging processing unit 22according to imaging information stored in the register group 27 andthus controls capturing an image in the imaging unit 21.

Incidentally, the register group 27 can store output control informationassociated with output control in the output control unit 23 in additionto imaging information and results of imaging signal processing in theimaging processing unit 22. The output control unit 23 can performoutput control of selectively outputting a captured image and a signalprocessing result according to the output control information stored inthe register group 27.

Further, in the imaging device 2, the imaging control unit 25 and a CPU31 of the signal processing block 30 are connected to each other via theconnection line CL1, and the CPU 31 can perform reading and writinginformation from and to the register group 27 via the connection lineCL1.

That is, in the imaging device 2, reading and writing of informationfrom and to the register group 27 can be performed by the communicationI/F 26 and can also be performed by the CPU 31.

The signal processing block 30 includes the central processing unit(CPU) 31, a digital signal processor (DSP) 32, a memory 33, acommunication I/F 34, the image compressing unit 35, and an input I/F36, and performs predetermined signal processing using a captured imageor the like acquired from the imaging block 10.

The CPU 31 to the input I/F 36 constituting the signal processing block30 are connected to each other via a bus and can perform exchange ofinformation if necessary.

The CPU 31 performs control of the signal processing block 30, readingand writing information from and to the register group 27 of the imagingcontrol unit 25 via the connection line CL1, and other processing byexecuting a program stored in the memory 33.

For example, by executing the program, the CPU 31 functions as animaging information calculating unit that calculates imaging informationusing a signal processing result acquired through signal processing inthe DSP 32, feeds back new imaging information calculated using thesignal processing result to the register group 27 of the imaging controlunit 25 via the connection line CL1, and stores the new imaginginformation therein.

Accordingly, the CPU 31 can control capturing an image in the imagingunit 21 or imaging signal processing in the imaging processing unit 22according to the signal processing result of the captured image.

Further, the imaging information which is stored in the register group27 by the CPU 31 can be provided (output) from the communication I/F 26to the outside. For example, information associated with a focus out ofthe imaging information stored in the register group 27 can be providedto a focus driver (not illustrated) that controls the focus from thecommunication I/F 26.

By executing a program stored in the memory 33, the DSP 32 functions asa signal processing unit that performs signal processing using acaptured image supplied from the imaging processing unit 22 to thesignal processing block 30 via the connection line CL2 or informationreceived from the outside by the input I/F 36.

The memory 33 includes a static random access memory (SRAM), a dynamicRAM (DRAM), or the like, and stores data necessary for processing in thesignal processing block 30 or the like.

For example, the memory 33 stores a program which is received from theoutside via the communication I/F 34, a captured image which iscompressed by the image compressing unit 35 and used for signalprocessing in the DSP 32, a signal processing result of the signalprocessing which is performed by the DSP 32, information which isreceived by the input I/F 36, and the like.

The communication I/F 34 is, for example, a second communication I/Fsuch as a serial communication I/F such as a serial peripheral interface(SPI), and exchange of necessary information such as a program which isexecuted by the CPU 31 or the DSP 32 with the outside (for example, thememory 3, the control unit 6, or the like in FIG. 1).

For example, the communication I/F 34 downloads a program which isexecuted by the CPU 31 or the DSP 32 from the outside, supplies theprogram to the memory 33, and stores the program therein.

Accordingly, the CPU 31 or the DSP 32 can perform various processesusing the program which is downloaded by the communication I/F 34.

Incidentally, the communication I/F 34 can perform exchange of arbitrarydata in addition to the program with the outside. For example, thecommunication I/F 34 can output a signal processing result which isacquired through signal processing in the DSP 32 to the outside.Further, the communication I/F 34 can output information based on aninstruction from the CPU 31 to an external device and thus control theexternal device in accordance with the instruction from the CPU 31.

Here, the signal processing result which is acquired through the signalprocessing in the DSP 32 can be output to the outside via thecommunication I/F 34 and can be written to the register group 27 of theimaging control unit 25 by the CPU 31. The signal processing resultwritten to the register group 27 can be output to the outside via thecommunication I/F 26. The same is true of a processing result ofprocessing which is performed by the CPU 31.

The image compressing unit 35 is supplied with a captured image from theimaging processing unit 22 via the connection line CL2. The imagecompressing unit 35 performs a compression process of compressing thecaptured image and generates a compressed image of which an amount ofdata is less than that of the captured image.

The compressed image generated by the image compressing unit 35 issupplied to the memory 33 via a bus and stored therein.

Here, signal processing in the DSP 32 can be performed using thecaptured image and can also be performed using the compressed imagegenerated from the captured image by the image compressing unit 35.Since the amount of data of the compressed image is less than that ofthe captured image, it is possible to achieve a decrease in load ofsignal processing in the DSP 32 or saving of a memory capacity of thememory 33 that stores the compressed image.

For example, scale-down of converting a captured image of 12M(3968×2976) pixels into an image with a VGA size can be performed as thecompression process which is performed by the image compressing unit 35.Further, in a case where the signal processing in the DSP 32 isperformed on luminance and the captured image is an RGB image, YUVconversion of converting the RGB image into, for example, a YUV imagecan be performed as the compression process.

Incidentally, the image compressing unit 35 may be embodied in softwareor may be embodied in dedicated hardware.

The input I/F 36 is an I/F that receives information from the outside.For example, the input I/F 36 receives an output of an external sensor(an external sensor output) from the external sensor, supplies theexternal sensor output to the memory 33 via a bus, and stores theexternal sensor output therein.

For example, similarly to the output I/F 24, a parallel I/F such as amobile industry processor interface (MIPI) or the like can be employedas the input I/F 36.

Further, for example, a range sensor that senses information associatedwith a distance can be employed as the external sensor. Furthermore, forexample, an image sensor that senses light and outputs an imagecorresponding to the light, that is, an image sensor which is differentfrom the imaging device 2, can be employed as the external sensor.

The DSP 32 can perform signal processing using the external sensoroutput which is received from the external sensor by the input I/F 36and stored in the memory 33 in addition to (a compressed image generatedfrom) a captured image.

In the single-chip imaging device 2 having the above-mentionedconfiguration, signal processing using (a compressed image generatedfrom) a captured image acquired through imaging in the imaging unit 21is performed by the DSP 32, and the signal processing result of thesignal processing and the captured image are selectively output from theoutput I/F 24. Accordingly, it is possible to construct an imagingdevice that outputs information required by a user with a small size.

Here, in a case where the imaging device 2 does not perform signalprocessing of the DSP 32 and the imaging device 2 does not output asignal processing result but outputs a captured image, that is, in acase where the imaging device 2 is configured as an image sensor thatonly captures and outputs an image, the imaging device 2 can beconstituted by only the imaging block 20 in which the output controlunit 23 is not provided.

FIG. 3 is a perspective view illustrating an example of an outlineconfiguration of the imaging device 2 illustrated in FIG. 1.

The imaging device 2 can be configured, for example, as a single-chipsemiconductor device having a stacked structure in which a plurality ofdies is stacked as illustrated in FIG. 3.

In FIG. 3, the imaging device 2 has a configuration in which two diesincluding dies 51 and 52 are stacked.

In FIG. 3, the imaging unit 21 is mounted in the upper die 51, and theimaging processing unit 22 to the imaging control unit 25 and the CPU 31to the input I/F 36 are mounted in the lower die 52.

The upper die 51 and the lower die 52 are electrically connected to eachother, for example, by forming a through-hole penetrating the die 51 andreaching the die 52 or by performing Cu—Cu bonding of directlyconnecting a Cu wire exposed from the bottom surface of the die 51 and aCu wire exposed from the top surface of the die 52.

Here, for example, a column-parallel AD system or an area AD system canbe employed as a system for performing AD conversion of an image signaloutput from the imaging unit 21 in the imaging processing unit 22.

In the column-parallel AD system, AD conversion of an image signal ofpixels in the columns of one row is performed I parallel, for example,by providing an AD converter (ADC) for each column of pixelsconstituting the imaging unit 21 and causing the ADC for each column totake charge of AD conversion of the pixel signals of the pixels in thecolumn. In a case where the column-parallel AD system is employed, apart of the imaging processing unit 22 that performs the AD conversionin the column-parallel AD system may be mounted in the upper die 51.

In the area AD system, pixels constituting the imaging unit 21 arepartitioned into a plurality of blocks and an ADC is provided for eachblock. In addition, AD conversion of the image signals of the pixels inthe plurality of blocks is performed in parallel by causing the ADC ofeach block to take charge of AD conversion of pixel signals of thepixels in the block. In the area AD system, AD conversion (reading andAD conversion) of an image signal can be performed on only necessarypixels out of the pixels constituting the imaging unit 21 with a blockas a minimum unit.

Incidentally, when an increase in area of the imaging device 2 ispermitted, the imaging device 2 can be constituted by one die.

Further, in FIG. 3, two dies 51 and 52 are stacked to constitute asingle-chip imaging device 2, but the single-chip imaging device 2 maybe constituted by stacking three or more dies. For example, in a casewhere three dies are stacked to constitute a single-chip imaging device2, the memory 33 illustrated in FIG. 3 may be mounted in a differentdie.

Here, in an imaging device (hereinafter also referred to as abump-connected imaging device) in which a sensor chip, a memory chip,and a DSP chip are connected in parallel using a plurality of bumps asdescribed in Patent Document 1, the thickness is larger than that of thesingle-chip imaging device 2 constituted in a stacked structure and thesize thereof is larger.

Furthermore, in the bump-connected imaging device, it may be difficultto secure a sufficient rate as a rate at which a captured image isoutput from the imaging processing unit 22 to the output control unit 23due to signal deterioration or the like in connection portions betweenthe bumps.

With the imaging device 2 having a stacked structure, it is possible toprevent an increase in size of the device or to prevent a sufficientrate from being secured as a rate between the imaging processing unit 22and the output control unit 23.

Accordingly, with the imaging device 2 having a stacked structure, it ispossible to achieve construction of an imaging device that outputsinformation required by a user with a small size.

In a case where the information required by a user is a captured image,the imaging device 2 can output the captured image.

Further, in a case where the information required by a user is acquiredthrough signal processing using a captured image, the imaging device 2can acquire and output a signal processing result as the informationrequired by the user by causing the DSP 32 to perform the signalprocessing.

For example, a recognition process of recognizing a predeterminedrecognition object from a captured image can be employed as the signalprocessing which is performed by the imaging device 2, that is, thesignal processing of the DSP 32.

Further, for example, the imaging device 2 can receive an output of arange sensor such as a time of flight (ToF) sensor which is arrangedwith a predetermined positional relationship with the imaging device 2via the input I/F 36. In this case, for example, a fusion process ofcombining the output of the range sensor and the captured image tocalculate a distance with high accuracy such as a process of removingnose of a range image acquired from the output of the range sensorreceived via the input I/F 36 using the captured image can be employedas the signal processing of the DSP 32.

Furthermore, for example, the imaging device 2 can receive an imageoutput from an image sensor, which is arranged to have a predeterminedpositional relationship with the imaging device 2, via the input I/F 36.In this case, for example, a self-localization process (simultaneouslylocalization and mapping (SLAM)) using an image received via the inputI/F 36 and a captured image as stereo images can be employed as thesignal processing of the DSP 32.

Hereinafter, processing of the imaging device 2 in a case where therecognition process, the fusion process, and the self-localizationprocess (hereinafter also referred to as an SLAM process) are performedas the signal processing of the DSP 32 will be described.

Incidentally, the order of processes which are performed the imagingdevice 2 which will be described below can be changed in a possiblerange. That is, the order of processes which are performed in theimaging device 2 is not limited to the following order.

Processing Example of Imaging Device 2 in Case where Recognition Processis Performed as Signal Processing in DSP 32

FIGS. 4, 5, and 6 are diagrams describing an outline of a processingexample of the imaging device 2 in a case where the recognition processis performed as signal processing of the DSP 32.

In Step S11 of FIG. 4, the communication I/F 34 downloads a program (acode) which is executed by the CPU 31 and the DSP 32 from the outsideand stores the program in the memory 33 in performing the recognitionprocess as the signal processing of the DSP 32. Here, the program whichis executed by the DSP 32 is a recognition processing program forperforming a recognition process as the signal processing.

The CPU 31 starts a predetermined process by executing a program storedin the memory 33.

That is, in Step S12, the CPU 31 reads (information of) brightness foreach sub area of a captured image and other necessary information fromthe register group 27 via the connection line CL1.

In Step S13, the CPU 31 performs control associated with the compressionprocess such as determining a reduction rate indicating a degree ofscaling-down of a captured image as the compression process in the imagecompressing unit 35.

In Step S14, the imaging unit 21 starts capturing of an image, and theimaging processing unit 22 starts outputting of an image from theimaging unit 21 as a captured image. Accordingly, supply of a capturedimage from the imaging processing unit 22 to the output control unit 23and supply of the captured image from the imaging processing unit 22 tothe image compressing unit 35 via the connection line CL2 are started.

The captured image supplied from the imaging processing unit 22 to theoutput control unit 23 is selected by the output control unit 23 ifnecessary and is output from the output I/F 24 to the outside.

In Step S15, the image compressing unit 35 starts a compression processof a captured image supplied from the imaging processing unit 22 via theconnection line CL2.

Here, an image output from the imaging unit 21 as well as an imageoutput from the image processing unit 22 is referred to as a capturedimage in the following description.

As described above with reference to FIG. 2, the imaging unit 21 canoutput a captured image of, for example, 12M pixels or a VGA size.Furthermore, the imaging unit 21 can output, for example, a color imageof RGB (red, green, and blue) or a monochrome image as a captured image.

In a case where a captured image is a full-size image of 12M pixels, theimage compressing unit 35 performs, for example, a process of scalingdown the captured image of 12M pixels into a captured image of a VGAsize or the like as the compression process.

Here, in a case where the captured image has a scaled-down size, thatis, an image of a VGA size herein, the image compressing unit 35 doesnot perform the scaling-down process.

Further, in a case where the captured image is a color image of RGB, theimage compressing unit 35 performs YUV conversion as the compressionprocess, for example, to convert a color captured image into amonochrome captured image.

Here, in a case where the captured image is a monochrome image, theimage compressing unit 35 does not perform YUV conversion.

Accordingly, for example, in a case where the captured image is a colorimage of 12M pixels, the image compressing unit 35 performs scaling-downand YUV conversion of the captured image as the compression process.

Further, for example, in a case where the captured image is a colorimage of a VGA size, the image compressing unit 35 performs YUVconversion of the captured image as the compression process.

The image compressing unit 35 stores the monochrome captured image of aVGA size acquired as the result of the compression result as acompressed image in the memory 33.

Incidentally, the imaging device 2 can be constituted without providingthe image compressing unit 35. Here, in a case where the imaging device2 is constituted without providing the image compressing unit 35, theload of the DSP 32 or the memory capacity required for the memory 33increases.

In Step S21 of FIG. 5, the DSP 32 starts a recognition process as signalprocessing corresponding to a recognition processing program by readingand executing the recognition processing program stored in the memory 33in Step S11.

That is, the DSP 32 sequentially reads areas of a compressed imagestored in the memory 33 as a recognition process target from the memory33 and performs a recognition process of recognizing a predeterminedrecognition object (for example, a face of a person or the like) fromthe processing target as signal processing using the compressed image(further the captured image).

The recognition process can be performed, for example, using a techniquesuch as deep learning of a convolutional neural network (CNN) or thelike. Further, in the recognition process, a specific subject such as aface of a person can be detected with the specific subject as arecognition target, and a scene appearing in the image can also bedetected with the scene appearing in the image as a recognition target.

In Step S22, the DSP 32 supplies the result of the recognition processas a signal processing result to the memory 33 and stores the result ofthe recognition process therein. The result of the recognition process(hereinafter also referred to as a recognition result) includes, forexample, information about whether a recognition target has beendetected, information of a detection position at which the recognitiontarget has been detected, or the like.

Incidentally, in the recognition process, gradation conversion of thecompressed image such as setting average brightness of the compressedimage to a predetermined fixed value can be performed in order toprevent luminance of a compressed image (a captured image) fromaffecting recognition accuracy. This gradation conversion can beperformed using brightness for each sub area of the captured image whichis read from the register group 27 by the CPU 31 in Step S12 of FIG. 4.

In Step S31 of FIG. 6, the CPU 31 reads a recognition result as a signalprocessing result stored in the memory 33 and performs an operation ofcalculating imaging information such as an exposure time suitable forcapturing a captured image using the recognition result.

For example, in a case where the recognition result includes a detectionposition of a face which is a recognition target, the CPU 31 calculatesan exposure time suitable for imaging a face appearing at the detectionposition according to the luminance or the like of the detectionposition of the face in the compressed image (the captured image).Further, for example, the CPU 31 calculates imaging information forcontrolling automatic focusing such that a focus is set on the detectionposition of the face.

In addition, the CPU 31 calculates imaging information such as a framerate, an imaging mode, and a cutting area suitable for capturing acaptured image using the recognition result if necessary.

In Step S32, the CPU 31 feeds back the imaging information calculated inStep S31 to the register group 27 via the connection line CL1. Theregister group 27 newly stores the imaging information fed back from theCPU 31 and then the imaging control unit 25 controls the imagingprocessing unit 22 on the basis of the imaging information newly storedin the register group 27.

In Step S33, the CPU 31 reads the recognition result as the signalprocessing result stored in the memory 33 and supplies the recognitionresult to the output control unit 23.

The recognition result as the signal processing result supplied from thememory 33 to the output control unit 23 is selected by the outputcontrol unit 23 if necessary and is output from the output I/F 24 to theoutside.

FIG. 7 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

For example, the imaging unit 21 captures an image of one frame for 1/60seconds in the first half of a frame period T1 with 1/30 seconds as theframe period T1. A captured image acquired through imaging of theimaging unit 21 is supplied from the imaging processing unit 22 to theoutput control unit 23 and the image compressing unit 35.

Incidentally, in FIG. 7, the captured image is assumed to be a colorimage of 12M pixels.

When the captured image is supplied from the imaging processing unit 22,the output control unit 23 selects the captured image and outputs theselected captured image from the output I/F 24 to the outside.

In the image compressing unit 35, scale-down and YUV conversion areperformed as the compression process of the color captured image of 12Mpixels, and the color captured image of 12M pixels is converted into amonochrome compressed image of a VGA size. This compressed image isstored in the memory 33.

Here, when a certain frame period T1 is noticed, the frame period T1 isalso referred to as a noticed frame period T1.

In FIG. 7, the compression process of the captured image in the noticedframe period T1 (captured for 1/60 seconds in the first half) ends inthe first half of the noticed frame period T1, and then the DSP 32starts a recognition process using the compressed image stored in thememory 33, that is, the compressed image acquired from the capturedimage in the noticed frame period T1 at a time at which 1/60 seconds inthe second half of the noticed frame period T1 starts.

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the recognition process using the compressed image acquired fromthe captured image in the noticed frame period T1 and supplies therecognition result of the recognition process as a signal processingresult to the output control unit 23.

When the recognition result as a signal processing result is supplied,the output control unit 23 selects the signal processing result andoutputs the signal processing result from the output I/F 24 to theoutside.

In FIG. 7, the signal processing result in the noticed frame period T1,that is, the signal processing result (the recognition result) of therecognition process using the compressed image acquired from thecaptured image in the noticed frame period T1, is output from the outputI/F 24 in a period from a time immediately before the noticed frameperiod T1 ends to the time at which the noticed frame period T1 ends.

FIG. 8 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

In FIG. 8, similarly to FIG. 7, the imaging unit 21 captures an image ofone frame for 1/60 seconds in the first half of a frame period T1 with1/30 seconds as the frame period T1 and a color captured image of 12Mpixels is acquired. The captured image acquired by the imaging unit 21is supplied from the imaging processing unit 22 to the output controlunit 23 and the image compressing unit 35.

Similarly to FIG. 7, when the captured image is supplied from theimaging processing unit 22, the output control unit 23 selects thecaptured image and outputs the selected captured image from the outputI/F 24 to the outside.

In the image compressing unit 35, similarly to FIG. 7, scale-down andYUV conversion are performed as the compression process of the colorcaptured image of 12M pixels, and the color captured image of 12M pixelsis converted into a monochrome compressed image of a VGA size. Thiscompressed image is stored in the memory 33.

In FIG. 8, similarly to FIG. 7, the compression process of the capturedimage in the noticed frame period T1 ends in the first half of thenoticed frame period T1.

Here, in FIG. 8, at a time at which a part of a compressed image whichis generated by the compression process can be used for the recognitionprocess without waiting of end of the compression process after thecompression process has started, the DSP 32 starts the recognitionprocess using (a part of) the compressed image acquired from thecaptured image in the noticed frame period T1.

Accordingly, in FIG. 8, the compression process and the recognitionprocess are performed in parallel in some period.

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the recognition process using the compressed image acquired fromthe captured image in the noticed frame period T1 and supplies therecognition result of the recognition process as a signal processingresult to the output control unit 23.

When the recognition result as a signal processing result is supplied,the output control unit 23 selects the signal processing result andoutputs the signal processing result from the output I/F 24 to theoutside.

In FIG. 8, similarly to FIG. 7, the recognition result of therecognition process using the compressed image acquired from thecaptured image in the noticed frame period T1 as the signal processingresult in the noticed frame period T1 is output from the output I/F 24in a period from a time immediately before the noticed frame period T1ends to the time at which the noticed frame period T1 ends.

As described above, in FIG. 8, since the recognition process is startedwithout waiting for end of the compression process, the time forperforming the recognition process can be secured longer in comparisonwith the case illustrated in FIG. 7 in which the recognition processstarts after the compression process has ended.

FIG. 9 is a timing diagram describing a third example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

In FIG. 9, similarly to FIG. 7, the imaging unit 21 captures an image ofone frame with 1/30 seconds as a frame period T1. Here, in FIG. 9, theimaging unit 21 captures a color captured image of a VGA size, not acolor captured image of 12M pixels. Accordingly, in FIG. 9, capturing ofan image in one frame ends for a very short time from start of the frameperiod T1. The captured image of a VGA size captured by the imaging unit21 is supplied from the imaging processing unit 22 to the output controlunit 23 and the image compressing unit 35.

Here, in FIG. 9, a captured image is not used in the outside, and thusthe output control unit 23 does not select the captured image and doesnot output the captured image from the output I/F 24 to the outside evenwhen the captured image is supplied from the imaging processing unit 22.

The image compressing unit 35 performs a compression process of thecaptured image and stores a compressed image acquired as a result in thememory 33.

Here, in FIG. 9, since the captured image is a color image of a VGAsize, YUV conversion is performed but scale-down is not performed as thecompression process. Accordingly, the compression process in FIG. 9 endsfor a shorter time in comparison with that in FIG. 7 or 8.

In addition, in FIG. 9, similarly to FIG. 8, at a time at which a partof a compressed image which is generated by the compression process canbe used for the recognition process without waiting of end of thecompression process after the compression process has started, the DSP32 starts the recognition process using (a part of) the compressed imageacquired from the captured image in the noticed frame period T1.

Accordingly, in FIG. 9, similarly to FIG. 8, the compression process andthe recognition process are performed in parallel in some period.

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the recognition process using the compressed image acquired fromthe captured image in the noticed frame period T1 and supplies therecognition result of the recognition process as a signal processingresult to the output control unit 23.

When the recognition result as a signal processing result is supplied,the output control unit 23 selects the signal processing result andoutputs the signal processing result from the output I/F 24 to theoutside.

In FIG. 9, similarly to FIG. 7 or 8, the recognition result of therecognition process using the compressed image acquired from thecaptured image in the noticed frame period T1 as the signal processingresult in the noticed frame period T1 is output from the output I/F 24in a period from a time immediately before the noticed frame period T1ends to the time at which the noticed frame period T1 ends.

In FIG. 9, similarly to FIG. 8, since the recognition process is startedwithout waiting for end of the compression process, the time forperforming the recognition process can be secured longer in comparisonwith the case illustrated in FIG. 7 in which the recognition processstarts after the compression process has ended.

Furthermore, in FIG. 9, since the captured image output from the imagingunit 21 is an image of a VGA size, scale-down may not be performed inthe compression process and thus it is possible to reduce a load of thecompression process.

As described above, the embodiment in which the captured image outputfrom the imaging unit 21 is not output as an image of a VGA size fromthe output I/F 24 is useful, for example, in a case where a capturedimage is not required by the outside and a signal processing result(herein the recognition result of the recognition process) is required.

FIG. 10 is a timing diagram describing a fourth example of a processingtime of the imaging device 2 in a case where a recognition process isperformed as signal processing of the DSP 32.

In FIG. 10, similarly to FIG. 9, the imaging unit 21 captures a colorcaptured image of a VGA size with 1/30 seconds as a frame period T1. Thecaptured image captured by the imaging unit 21 is supplied from theimaging processing unit 22 to the output control unit 23 and the imagecompressing unit 35.

Here, in FIG. 10, similarly to FIG. 9, a captured image is not used inthe outside, and thus the output control unit 23 does not output (doesnot select) the captured image from the output I/F 24 to the outside.

The image compressing unit 35 performs a compression process of thecaptured image and stores a compressed image acquired as a result in thememory 33.

Here, in FIG. 10, similarly to FIG. 9, since YUV conversion is performedbut scale-down is not performed as the compression process, thecompression process ends for a shorter time.

In addition, in FIG. 10, similarly to FIG. 8 or 9, at a time at which apart of a compressed image which is generated by the compression processcan be used for the recognition process without waiting of end of thecompression process after the compression process has started, the DSP32 starts the recognition process using (a part of) the compressed imageacquired from the captured image in the noticed frame period T1.

Accordingly, in FIG. 10, similarly to FIG. 8 or 9, the compressionprocess and the recognition process are performed in parallel in someperiod.

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the recognition process using the compressed image acquired fromthe captured image in the noticed frame period T1 and supplies therecognition result of the recognition process as a signal processingresult to the output control unit 23.

Similarly to FIGS. 7 to 9, the output control unit 23 selects therecognition result using the compressed image acquired from the capturedimage in the noticed frame period T1 as the signal processing result inthe noticed frame period T1, and outputs the recognition result from theoutput I/F 24 immediately before the noticed frame period T1 ends.

In FIG. 10, the DSP 32 appropriately outputs intermediate data acquiredin the middle of the recognition process while the recognition processas the signal processing is being performed. The intermediate dataoutput from the DSP 32 is supplied to the output control unit 23, andthe output control unit 23 selects the intermediate data and outputs theselected intermediate data from the output I/F 24 similarly to thesignal processing result when the intermediate data is supplied.

As described above, in a case where the intermediate data in the middleof the signal processing (herein the recognition process) is output fromthe output I/F 24 to the outside, the intermediate data can be providedfor debugging of the program performing the signal processing (hereinthe recognition processing program).

Processing Example of Imaging Device 2 in Case where Fusion Process isPerformed as Signal Processing in DSP 32

FIGS. 11, 12, and 13 are diagrams illustrating an outline of aprocessing example of the imaging device 2 in a case where the fusionprocess is performed as signal processing of the DSP 32.

In Step S41 of FIG. 11, the communication I/F 34 downloads a programwhich is executed by the CPU 31 and the DSP 32 from the outside andstores the program in the memory 33 in performing the fusion process asthe signal processing of the DSP 32. Here, the program which is executedby the DSP 32 is a fusion processing program for performing a fusionprocess as the signal processing.

The CPU 31 starts a predetermined process by executing a program storedin the memory 33.

That is, in Step S42, the CPU 31 reads (information of) brightness foreach sub area of a captured image and other necessary information fromthe register group 27 via the connection line CL1.

In Step S43, the CPU 31 performs control associated with the compressionprocess such as determining a reduction rate indicating a degree ofscaling-down of a captured image as the compression process in the imagecompressing unit 35.

In Step S44, the imaging unit 21 starts capturing of an image, and theimaging processing unit 22 starts outputting of a captured image fromthe imaging unit 21. Accordingly, supply of a captured image from theimaging processing unit 22 to the output control unit 23 and supply ofthe captured image from the imaging processing unit 22 to the imagecompressing unit 35 via the connection line CL2 are started.

The captured image supplied from the imaging processing unit 22 to theoutput control unit 23 is selected by the output control unit 23 ifnecessary and is output from the output I/F 24 to the outside.

In Step S45, the image compressing unit 35 starts a compression processof a captured image supplied from the imaging processing unit 22 via theconnection line CL2.

The image compressing unit 35 stores the monochrome captured image of aVGA size acquired as the result of the compression result as acompressed image in the memory 33.

In a case where the fusion process is performed as the signalprocessing, a sensor output of a range sensor, for example, a ToF sensor(not illustrated) installed to satisfy a predetermined positionalrelationship with the imaging device 2, is supplied to the input I/F 36from the ToF sensor.

Here, the sensor output of the ToF sensor has a form of an image having,for example, a sensing result of a distance (for example, a valuecorresponding to the time until light emitted from the ToF sensor isreflected from a subject and is received by the ToF sensor or the like)as a pixel value. Hereinafter, this image is also referred to as a ToFimage. The ToF image is, for example, an image of a QQVGA size or a QVGAsize.

In Step S46, the input I/F 36 starts receiving the ToF image as thesensor output of the ToF sensor. The ToF image received by the input I/F36 is supplied to the memory 33 and stored therein.

In Step S51 of FIG. 12, the DSP 32 starts a fusion process as signalprocessing corresponding to a fusion processing program by reading andexecuting the fusion processing program stored in the memory 33 in StepS41.

That is, the DSP 32 sequentially reads areas of a compressed imagestored in the memory 33 as a fusion process target from the memory 33,reads the ToF image from the memory 33, and performs a fusion processusing the processing target of the compressed image and the ToF image.

In the fusion process, for example, the sensor output which is pixelvalues of the ToF image is converted into a value indicating a distance,and a range image with the value indicating the distance as pixel valuesis generated. In this embodiment, for example, one range image isacquired from four continuous ToF images.

Furthermore, in the fusion process, calibration of aligning the pixelsof (the processing target of) the compressed image and the correspondingpixels of the range image is performed, for example, on the basis of thepositional relationship between the imaging device 2 and the ToF sensor.

Further, in the fusion process, noise of the range image is removed, forexample, by referring to the texture or the like of the compressedimages and performing correction or the like of matching valuesindicating a distance as pixel values of a plurality of pixels of arange image corresponding to a plurality of pixels in the compressedimages in which a subject at an equidistance appears.

In Step S52, the DSP 32 supplies the result of the fusion process as asignal processing result to the memory 33 and stores the result of thefusion process therein. The result of the fusion process is, forexample, a range image from which noise has been removed.

Incidentally, in the fusion process, gradation conversion of thecompressed image such as setting average brightness of the compressedimage to a predetermined fixed value can be performed in order toprevent luminance of a compressed image (a captured image) fromaffecting removal of noise of the range image. This gradation conversioncan be performed using brightness for each sub area of the capturedimage which is read from the register group 27 by the CPU 31 in Step S42of FIG. 11.

In Step S61 of FIG. 13, the CPU 31 reads a range image as a signalprocessing result stored in the memory 33 and performs an operation ofcalculating imaging information such as focusing suitable for capturinga captured image using the range image.

For example, the CPU 31 detects a closest subject or a subject distantat a predetermined distance from the range image and calculates imaginginformation for controlling automatic focusing to set a focus on thedetected subject. Further, for example, the CPU 31 detects a closestsubject or a subject distant at a predetermined distance from the rangeimage and calculates an exposure time suitable for imaging the detectedsubject according to the luminance or the like of the subject.

In addition, the CPU 31 calculates imaging information such as a framerate, an imaging mode, and a cutting area suitable for capturing acaptured image using the range image if necessary.

In Step S62, the CPU 31 feeds back the imaging information calculated inStep S61 to the register group 27 via the connection line CL1. Theregister group 27 newly stores the imaging information fed back from theCPU 31 and then the imaging control unit 25 controls the imagingprocessing unit 22 on the basis of the imaging information newly storedin the register group 27.

In Step S63, the CPU 31 reads the range image as the signal processingresult stored in the memory 33 and supplies the range image to theoutput control unit 23.

The range image as the signal processing result supplied from the memory33 to the output control unit 23 is selected by the output control unit23 if necessary and is output from the output I/F 24 to the outside.

FIG. 14 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where a fusion process isperformed as signal processing of the DSP 32.

For example, the imaging unit 21 captures a color captured image of 12Mpixels for 1/60 seconds in the first half of a frame period T1 with 1/30seconds as the frame period T1. A captured image acquired throughimaging of the imaging unit 21 is supplied from the imaging processingunit 22 to the output control unit 23 and the image compressing unit 35.

When the captured image is supplied from the imaging processing unit 22,the output control unit 23 selects the captured image and outputs theselected captured image from the output I/F 24 to the outside.

In the image compressing unit 35, scale-down and YUV conversion areperformed as the compression process of the color captured image of 12Mpixels, and the color captured image of 12M pixels is converted into amonochrome compressed image of a VGA size. This compressed image isstored in the memory 33.

In FIG. 14, a ToF sensor is connected to the input I/F 36 of the imagingdevice 2, and the ToF sensor outputs a ToF image of a QVGA size as asensor output.

The input I/F 36 receives the ToF image as the sensor output of the ToFsensor and stores the ToF image in the memory 33.

Here, in FIG. 14, the ToF sensor can output a ToF image of a QVGA sizeat 240 fps (frames per second). In FIG. 14, the ToF sensor outputs onlyfour ToF images of a QVGA size at 240 fps for 1/60 seconds in the firsthalf of the frame period T1, and the input I/F 36 receives the four ToFimages for 1/60 seconds in the first half of the frame period T1.

In FIG. 14, the compression process of the captured image in the noticedframe period T1 (captured for 1/60 seconds in the first half) ends inthe first half of the noticed frame period T1. Furthermore, in the firsthalf of the noticed frame period T1, receiving of four ToF images by theinput I/F 36 ends.

Thereafter, at a time at which 1/60 seconds in the second half of thenoticed frame period T1 starts, the DSP 32 starts the fusion processusing the compressed image stored in the memory 33, that is, thecompressed image acquired from the captured image in the noticed frameperiod T1, and the four ToF images stored in the memory 33.

In the fusion process, for example, one range image in the noticed frameperiod T1 is generated from the four ToF images in the noticed frameperiod T1, and calibration of aligning the pixels of the compressedimage and the pixels of the range image is performed. Then, noise of therange image is removed using the compressed image.

Here, in FIG. 14, the frame rate of the ToF images is 240 fps, and theframe rate of the range image is 30 fps corresponding to the frameperiod T1 ( 1/30 seconds).

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the fusion process using the compressed image acquired from thecaptured image in the noticed frame period T1 and supplies the rangeimage from which noise has been removed and which has been acquired asthe result of the fusion process as a signal processing result to theoutput control unit 23.

When the range image as a signal processing result is supplied, theoutput control unit 23 selects the signal processing result and outputsthe signal processing result from the output I/F 24 to the outside.

In FIG. 14, the signal processing result in the noticed frame period T1,that is, the signal processing result (the range image) of the fusionprocess using the compressed image acquired from the captured image inthe noticed frame period T1, is output from the output I/F 24 in aperiod from a time immediately before the noticed frame period T1 endsto the time at which the noticed frame period T1 ends.

FIG. 15 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where a fusion process isperformed as signal processing of the DSP 32.

In FIG. 15, similarly to FIG. 14, the imaging unit 21 captures a colorcaptured image of 12M pixels for 1/60 seconds in the first half of aframe period T1 with 1/30 seconds as the frame period T1 and thecaptured image is output from the output I/F 24 to the outside.

Furthermore, in FIG. 15, similarly to FIG. 14, the image compressingunit 35 generates a monochrome compressed image of a VGA size byscale-down and YUV conversion as the compression process of the colorcaptured image of 12M pixels, and stores the generated monochromecompressed image in the memory 33.

Further, in FIG. 15, similarly to FIG. 14, a ToF sensor is connected tothe input I/F 36 of the imaging device 2, and the ToF sensor outputs aToF image of a QVGA size as a sensor output.

Here, in FIG. 15, the ToF sensor outputs a ToF image of a QVGA size at120 fps. Accordingly, in FIG. 15, the time required for the ToF sensorto output four ToF images required for generating one range image is1/30 seconds= 1/120×4 seconds, that is, equal to the frame period T1.

The input I/F 36 receives the ToF images as the sensor output of the ToFsensor and stores the ToF images in the memory 33.

That is, in FIG. 15, since the ToF sensor outputs four ToF images of aQVGA size at 120 fps in the frame period T1 as described above, theinput I/F 36 receives the four ToF images in the frame period T1.

Accordingly, in receiving of the ToF images which is started at thestart of the noticed frame period T1, receiving of four ToF imagesrequired for generating a range image is completed (almost) at the endof the noticed frame period T1.

Accordingly, it is difficult to complete the fusion process using thecompressed image acquired from the captured image in the noticed frameperiod T1 and four ToF images received from the ToF sensor in thenoticed frame period T1 within the noticed frame period T1.

Therefore, in FIG. 15, the DSP 32 starts the fusion process using thecompressed image acquired from the captured image in the noticed frameperiod T1 and four ToF images received from the ToF sensor in thenoticed frame period T1 at the start time of a next frame period T1 ofthe noticed frame period T1.

Hereinafter, the compressed image acquired from the captured image inthe noticed frame period T1 is also referred to as a compressed image ofthe noticed frame period T1, and (four) ToF images received from the ToFsensor in the noticed frame period T1 is also referred to as (four) ToFimages of the noticed frame period T1.

The DSP 32 starts the fusion process using the compressed image of thenoticed frame period T1 and four ToF images of the noticed frame periodT1 at the start time of the next frame period T1 of the noticed frameperiod T1, and ends the fusion process at a time slightly before thefirst half of the next frame period T1 of the noticed frame period T1ends.

Then, the DSP 32 supplies the range image from which noise has beenremoved and which has been acquired as the result of the fusion processas a signal processing result to the output control unit 23.

Here, the signal processing result of the fusion process using thecompressed image of the noticed frame period T1 and four ToF images ofthe noticed frame period T1 and the range image as the signal processingresult are also referred to as a signal processing result of the noticedframe period T1 and the range image of the noticed frame period T1.

After outputting of the captured image in the next frame period T1 ofthe noticed frame period T1 from the output I/F 24 has ended, the outputcontrol unit 23 selects the range image as the signal processing resultof the noticed frame period T1 and outputs the selected range image fromthe output I/F 24 to the outside.

Accordingly, in FIG. 15, similarly to FIG. 14, the frame rate of thedistance image is 30 fps which corresponds to the frame period T1 ( 1/30seconds), and the range image as the signal processing result of thenoticed frame period T1 is not output in the noticed frame period T1 butis output in the next frame period T1.

In FIG. 14, the range image as the signal processing result of thenoticed frame period T1 is output in the noticed frame period T1. On theother hand, in FIG. 15, the range image as the signal processing resultof the noticed frame period T1 is output in the next frame period T1 ofthe noticed frame period T1. Accordingly, in FIG. 15, a ToF sensor inwhich the frame rate of a ToF image is lower than that in the case ofFIG. 14, that is, a ow-cost ToF sensor, can be used as the ToF sensorconnected to the input I/F 36.

Incidentally, as described above with reference to FIGS. 11 to 15, aform of use of the imaging device 2 in which a sensor output of a rangesensor such as a ToF sensor is received from the input I/F 36 and thefusion process is performed can be applied, for example, to automaticdriving of a vehicle and the like.

Processing Example of Imaging Device 2 in Case where SLAM Process isPerformed as Signal Processing in DSP 32

FIGS. 16, 17, and 18 are diagrams illustrating an outline of aprocessing example of the imaging device 2 in a case where the SLAMprocess is performed as signal processing of the DSP 32.

In Step S71 of FIG. 16, the communication I/F 34 downloads a programwhich is executed by the CPU 31 and the DSP 32 from the outside andstores the program in the memory 33 in performing the SLAM process asthe signal processing of the DSP 32. Here, the program which is executedby the DSP 32 is an SLAM processing program for performing an SLAMprocess as the signal processing.

The CPU 31 starts a predetermined process by executing a program storedin the memory 33.

That is, in Step S72, the CPU 31 reads (information of) brightness foreach sub area of a captured image and other necessary information fromthe register group 27 via the connection line CL1.

In Step S73, the CPU 31 performs control associated with the compressionprocess such as determining a reduction rate indicating a degree ofscaling-down of a captured image as the compression process in the imagecompressing unit 35.

In Step S74, the imaging unit 21 starts capturing of an image, and theimaging processing unit 22 starts outputting of a captured image fromthe imaging unit 21. Accordingly, supply of a captured image from theimaging processing unit 22 to the output control unit 23 and supply ofthe captured image from the imaging processing unit 22 to the imagecompressing unit 35 via the connection line CL2 are started.

The captured image supplied from the imaging processing unit 22 to theoutput control unit 23 is selected by the output control unit 23 ifnecessary and is output from the output I/F 24 to the outside.

In Step S75, the image compressing unit 35 starts a compression processof a captured image supplied from the imaging processing unit 22 via theconnection line CL2.

The image compressing unit 35 stores the monochrome captured image of aVGA size acquired as the result of the compression result as acompressed image in the memory 33.

In a case where the SLAM process is performed as the signal processing,a sensor output of an image sensor (not illustrated) installed tosatisfy a predetermined positional relationship with the imaging device2, is supplied to the input I/F 36 from the image sensor.

Hereinafter, the image sensor which is installed to satisfy apredetermined positioning relationship with the imaging device 2 andwhich is different from the imaging device 2 is also referred to asother sensor. Furthermore, the other sensor senses light and outputs animage corresponding to the light as a sensor output, and the image asthe sensor output of the other sensor is also referred to as othersensor image. Here, it is assumed that the other sensor image is, forexample, an image of a VGA size.

In Step S76, the input I/F 36 starts receiving the other sensor image ofa VGA size as the sensor output of the other sensor. The other sensorimage of a VGA size received by the input I/F 36 is supplied to thememory 33 and stored therein.

In Step S81 of FIG. 17, the DSP 32 starts an SLAM process as signalprocessing corresponding to an SLAM processing program by reading andexecuting the SLAM processing program stored in the memory 33 in StepS71.

That is, the DSP 32 sequentially reads areas of a compressed imagestored in the memory 33 as a processing target of the SLAM process fromthe memory 33, reads the other sensor image from the memory 33, andperforms the SLAM process using the processing target of the compressedimage and the other sensor image as stereoscopic images.

In the SLAM process, rectification of aligning the compressed image andthe other sensor image to be in parallel to each other (horizontallyequating the imaging device 2 and the other sensor) is performed, forexample, on the basis of the positional relationship between the imagingdevice 2 and the other sensor.

In addition, in the SLAM process, self-localization and construction(development) of a map are performed, for example, using the compressedimage and the other sensor image as stereoscopic images having beensubjected to the rectification.

In Step S82, the DSP 32 supplies the result of the SLAM process as asignal processing result to the memory 33 and stores the result of theSLAM process therein. The result of the SLAM process is, for example, alocalization result of self-localization (hereinafter also referred toas a localization result) and a map constructed along with theself-localization.

Incidentally, in the SLAM process, gradation conversion of thecompressed image and the other sensor image such as setting averagebrightness of the compressed image and the other sensor image to apredetermined fixed value can be performed in order to prevent luminanceof the compressed image (a captured image) and the other sensor imagefrom affecting self-localization or construction of a map. Thisgradation conversion can be performed using brightness for each sub areaof the captured image which is read from the register group 27 by theCPU 31 in Step S72 of FIG. 16.

In Step S91 of FIG. 18, the CPU 31 reads the localization result and themap as a signal processing result stored in the memory 33 and performsan operation of calculating imaging information such as an exposuretime, focusing, a frame rate, an imaging mode, and a cutting areasuitable for capturing a captured image using the localization resultand the map, if necessary.

In Step S92, the CPU 31 feeds back the imaging information calculated inStep S91 to the register group 27 via the connection line CL1. Theregister group 27 newly stores the imaging information fed back from theCPU 31 and then the imaging control unit 25 controls the imagingprocessing unit 22 on the basis of the imaging information newly storedin the register group 27.

In Step S93, the CPU 31 reads the localization result and the map as thesignal processing result stored in the memory 33 and supplies the rangeimage to the output control unit 23.

The localization result and the map as the signal processing resultsupplied from the memory 33 to the output control unit 23 are selectedby the output control unit 23 and are output from the output I/F 24 tothe outside.

FIG. 19 is a timing diagram describing a first example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

For example, the imaging unit 21 captures a color captured image of 12Mpixels for 1/60 seconds in the first half of a frame period T1 with 1/30seconds as the frame period T1. A captured image acquired throughimaging of the imaging unit 21 is supplied from the imaging processingunit 22 to the output control unit 23 and the image compressing unit 35.

When the captured image is supplied from the imaging processing unit 22,the output control unit 23 selects the captured image and outputs theselected captured image from the output I/F 24 to the outside.

In the image compressing unit 35, scale-down and YUV conversion areperformed as the compression process of the color captured image of 12Mpixels, and the color captured image of 12M pixels is converted into amonochrome compressed image of a VGA size. This compressed image isstored in the memory 33.

In FIG. 19, another sensor is connected to the input I/F 36 of theimaging device 2, and the other sensor outputs the other sensor image ofa VGA size as a sensor output.

The input I/F 36 receives the other sensor image as the sensor output ofthe other sensor and stores the other sensor image in the memory 33.

Here, in FIG. 19, the other sensor outputs the other sensor image of aVGA size at 30 fps which is the same as the frame period T1. That is, inFIG. 19, the other sensor outputs the other sensor image of a VGA sizeat 30 fps in synchronization with the imaging device 2 at the time ofstart of the frame period T1. The input I/F 36 receives the other sensorimage.

In FIG. 19, the compression process of the captured image in the noticedframe period T1 ends in the first half of the noticed frame period T1.

Thereafter, at a time at which 1/60 seconds in the second half of thenoticed frame period T1 starts, the DSP 32 starts the SLAM process usingthe compressed image acquired from the captured image in the noticedframe period T1 stored in the memory 33 and the other sensor image inthe noticed frame period T1 stored in the memory 33.

In the SLAM process, for example, rectification of a compressed image(of a captured image) in the noticed frame period T1 and the othersensor image in the noticed frame period T1 is performed andself-localization and construction of a map in the noticed frame periodT1 are performed using the compressed image and the other sensor imagehaving been subjected to the rectification.

At a time slightly before the noticed frame period T1 ends, the DSP 32ends the SLAM process using the compressed image and the other sensorimage in the noticed frame period T1 and supplies the localizationresult and the map which have been acquired as the result of the SLAMprocess as a signal processing result to the output control unit 23.

When the localization result and the map as a signal processing resultare supplied, the output control unit 23 selects the signal processingresult and outputs the signal processing result from the output I/F 24to the outside.

In FIG. 19, the signal processing result in the noticed frame period T1,that is, the signal processing result (the localization result and themap) of the SLAM process using the compressed image and the other sensorimage in the noticed frame period T1, is output from the output I/F 24in a period from the time immediately before the noticed frame period T1ends to the time at which the noticed frame period T1 ends.

FIG. 20 is a timing diagram describing a second example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

In FIG. 20, similarly to FIG. 19, the imaging unit 21 captures a colorcaptured image of 12M pixels with 1/30 seconds as a frame period T1 andoutputs the color captured image from the output I/F 24 to the outside.

Furthermore, in FIG. 20, similarly to FIG. 19, the image compressingunit 35 generates a monochrome compressed image of a VGA size byperforming scale-down and YUV conversion as the compression process ofthe color captured image of 12M pixels, and stores the color capturedimage of 12M pixels in the memory 33.

Further, in FIG. 20, similarly to FIG. 19, another sensor is connectedto the input I/F 36 of the imaging device 2, and the other sensoroutputs the other sensor image of a VGA size as a sensor output.

Similarly to FIG. 19, the input I/F 36 receives the other sensor imageas the sensor output of the other sensor and stores the other sensorimage in the memory 33.

In FIG. 20, similarly to FIG. 19, the compression process of thecaptured image in the noticed frame period T1 ends in the first half ofthe noticed frame period T1.

Thereafter, at a time at which 1/60 seconds in the second half of thenoticed frame period T1 starts, the DSP 32 starts the SLAM process usingthe compressed image acquired from the captured image in the noticedframe period T1 stored in the memory 33 and the other sensor image inthe noticed frame period T1 stored in the memory 33.

In the SLAM process, for example, rectification of a compressed image inthe noticed frame period T1 and the other sensor image in the noticedframe period T1 is performed and self-localization and construction of amap in the noticed frame period T1 are performed using the compressedimage and the other sensor image having been subjected to therectification.

In FIG. 20, at a time slightly before the first half of a next frameperiod T1 of the noticed frame period T1 ends, the DSP 32 ends the SLAMprocess using the compressed image and the other sensor image in thenoticed frame period T1 and supplies the localization result and the mapwhich have been acquired as the result of the SLAM process as a signalprocessing result to the output control unit 23.

After outputting of the captured image in the next frame period T1 ofthe noticed frame period T1 from the output I/F 24 has ended, the outputcontrol unit 23 selects the localization result and the map as thesignal processing result in the noticed frame period T1 and outputs thelocalization result and the map from the output I/F 24 to the outside.

Accordingly, in FIG. 20, the localization result and the map as thesignal processing result in the noticed frame period T1 are not outputin the noticed frame period T1 but are output in the next frame periodT1.

The localization result and the map as the signal processing result inthe noticed frame period T1 are output in the noticed frame period T1 inFIG. 19, but the localization result and the map as the signalprocessing result in the noticed frame period T1 are output in the nextframe period T1 of the noticed frame period T1 in FIG. 20. Accordingly,in FIG. 20, a longer time can be assigned to the SRAM process incomparison with the case illustrated in FIG. 19. As a result, it ispossible to improve accuracy of the localization result and the map asthe signal processing result of the SRAM process.

FIG. 21 is a timing diagram describing a third example of a processingtime of the imaging device 2 in a case where an SLAM process isperformed as signal processing of the DSP 32.

In FIG. 21, the imaging unit 21 captures an image of one frame with 1/30seconds as a frame period T1. Here, in FIG. 21, the imaging unit 21captures a color captured image of a VGA size, not a color capturedimage of 12M pixels. Accordingly, in FIG. 21, capturing of an image inone frame ends for a very short time from start of the frame period T1.The captured image captured by the imaging unit 21 is supplied from theimaging processing unit 22 to the output control unit 23 and the imagecompressing unit 35.

Here, in FIG. 21, a captured image is not used in the outside, and thusthe output control unit 23 does not select the captured image and doesnot output the captured image from the output I/F 24 to the outside evenwhen the captured image is supplied from the imaging processing unit 22.

The image compressing unit 35 performs a compression process of thecaptured image and stores a compressed image acquired as a result in thememory 33.

Here, in FIG. 21, since the captured image is a color image of a VGAsize, YUV conversion is performed but scale-down is not performed as thecompression process. Accordingly, the compression process in FIG. 21ends for a shorter time in comparison with a case where the capturedimage is a color image of 12M pixels.

In FIG. 21, similarly to FIG. 19 or 20, another sensor is connected tothe input I/F 36 of the imaging device 2, and the other sensor outputsthe other sensor image of a VGA size at 30 fps as a sensor output.

The input I/F 36 receives the other sensor image as the sensor output ofthe other sensor and stores the other sensor image in the memory 33.

In FIG. 21, when the other sensor image in the noticed frame period T1is stored in the memory 33, the compression process of the capturedimage in the noticed frame period T1 ends, and the compressed image inthe noticed frame period T1 acquired by the compression process isstored in the memory 33.

That is, when the other sensor image in the noticed frame period T1 isstored in the memory 33, both the compressed image and the other sensorimage in the noticed frame period T1 are stored in the memory 33 and theSLAM process using the compressed image and the other sensor image canbe started.

Thereafter, the DSP 32 starts the SLAM process using the compressedimage and the other sensor image in the noticed frame period T1 storedin the memory 33 as stereoscopic images.

In the SLAM process, for example, rectification of a compressed image inthe noticed frame period T1 and the other sensor image in the noticedframe period T1 is performed and self-localization and construction of amap in the noticed frame period T1 are performed using the compressedimage and the other sensor image having been subjected to therectification.

In FIG. 21, at a time slightly before the noticed frame period T1 ends,the DSP 32 ends the SLAM process using the compressed image and theother sensor image in the noticed frame period T1 and supplies thelocalization result and the map which have been acquired as the resultof the SLAM process as a signal processing result to the output controlunit 23.

When the localization result and the map as a signal processing resultare supplied, the output control unit 23 selects the signal processingresult and outputs the signal processing result from the output I/F 24to the outside.

In FIG. 21, the signal processing result in the noticed frame period T1,that is, the signal processing result (the localization result and themap) of the SLAM process using the compressed image and the other sensorimage in the noticed frame period T1, is output from the output I/F 24in a period from the time immediately before the noticed frame period T1ends to the time at which the noticed frame period T1 ends.

Furthermore, in FIG. 21, since the captured image output from theimaging unit 21 is an image of a VGA size, scale-down may not beperformed in the compression process and thus it is possible to reduce aload of the compression process.

As described above, the embodiment in which the captured image outputfrom the imaging unit 21 is not output as an image of a VGA size fromthe output I/F 24 is useful, for example, in a case where a capturedimage is not required by the outside and a signal processing result(herein the signal processing result of the SLAM process) is required.

Incidentally, as described above with reference to FIGS. 16 to 21, aform of use of the imaging device 2 in which the other sensor image isreceived from the input I/F 36 and the SLAM process is performed can beapplied, for example, to robot which moves autonomously and the like.

Here, in a case where the input I/F 36 receives the other sensor imageof the other sensor and the other sensor image and (a compressed imagegenerated from) a captured image captured by the imaging device 2 areused as stereoscopic images, rectification is necessary.

In FIGS. 16 to 21, rectification is performed as a part of the SLAMprocess which is performed by causing the DSP 32 to execute an SLAMprocessing program, that is, rectification is performed in software, butnecessary rectification cannot be performed in software but can beperformed in dedicated hardware which is provided in the imaging device2 in a case where the other sensor image and the captured image are usedas stereoscopic images.

Another Example of Configuration of Imaging Device 2

FIG. 22 is a block diagram illustrating another example of theconfiguration of the imaging device 2 illustrated in FIG. 1.

That is, FIG. 22 illustrates an example of a configuration of theimaging device 2 in which dedicated hardware for rectification isprovided.

Incidentally, in the drawing, elements corresponding to those in FIG. 2will be referred to by the same reference signs and description thereofwill be omitted in the following description.

In FIG. 22, the imaging device 2 includes the imaging unit 21 to theimaging control unit 25, the CPU 31 to the input I/F 36, and arectification unit 71.

Accordingly, the imaging device 2 illustrated in FIG. 22 is similar tothat illustrated in FIG. 2, in that the imaging unit 21 to the imagingcontrol unit 25 and the CPU 31 to the input I/F 36 are provided.

However, the imaging device 2 illustrated in FIG. 22 is different fromthat illustrated in FIG. 2, in that the rectification unit 71 is newlyprovided.

The rectification unit 71 is dedicated hardware for rectification andperforms rectification on a compressed image and the other sensor imagestored in the memory 33.

In FIG. 22, the DSP 32 performs an SLAM process using the compressedimage and the other sensor image having been subjected to rectificationby the rectification unit 71.

As described above, by providing the rectification unit 71 as dedicatedhardware for rectification, it is possible to increase the speed ofrectification.

Example of Use of Imaging Device 2

FIG. 23 is a diagram illustrating an example of use of the imagingdevice 2 illustrated in FIG. 1.

The imaging device 2 can be used, for example, for various electronicdevices that sense light such as visible light, infrared light,ultraviolet light, and X-rays as will be described below.

An electronic device that captures an image which is provided forappreciation such as a digital camera or a portable device having acamera function

An electronic device that is provided for traffic such as an onboardsensor that images the rear, the surroundings, the vehicle interior, andthe like of a vehicle, a monitoring camera that monitors travelingvehicles or a road, and a distance sensor that measures an inter-vehicledistance or the like for the purpose of safe driving such as automaticstop, recognition of a driver's state, and the like

An electronic device that is provided for home appliance such as a TV, arefrigerator, or an air conditioner to image a user's gesture and tooperate a device according to the gesture

An electronic device that is provided for medical purpose or health caresuch as an endoscope, an electronic microscope, or a device that imagesa blood vessel by receiving infrared light

An electronic device that is provided for security such as a monitoringcamera for crime prevention or a camera for authenticating a person

An electronic device that is provided for beauty treatment such as askin meter that images the skin and a microscope that images the scalp

An electronic device that is provided for sports such as a dedicatedaction camera for sports or the like or a wearable camera

An electronic device that is provided for agriculture such as a camerafor monitoring states of farms or crops

Application to Moving Object

The technology according to the present disclosure (the presenttechnology) can be applied to various products. For example, thetechnology according to the present disclosure may be implemented as adevice which is mounted in any kind of moving object such as a vehicle,an electric vehicle, a hybrid electric vehicle, a motor bike, a bicycle,a personal mobility, an airplane, a drone, a ship, and a robot.

FIG. 24 is a block diagram schematically illustrating an example of aconfiguration of a vehicle control system as an example of a movingobject control system to which the technology according to the presentdisclosure can be applied.

A vehicle control system 12000 includes a plurality of electroniccontrol units which are connected via a communication network 12001. Inthe example illustrated in FIG. 24, the vehicle control system 12000includes a drive system control unit 12010, a body system control unit12020, an exterior information detecting unit 12030, an interiorinformation detecting unit 12040, and a combined control unit 12050.Further, as functional elements of the combined control unit 12050, amicrocomputer 12051, a voice and image output unit 12052, and an onboardnetwork interface (I/F) 12053 are illustrated.

The drive system control unit 12010 controls operations of devicesassociated with a drive system of a vehicle in accordance with variousprograms. For example, the drive system control unit 12010 functions asa controller for a driving force generating device that generates adriving force for the vehicle such as an internal combustion engine or adrive motor, a driving force transmission mechanism that transmits adriving force to vehicle wheels, a steering mechanism that adjusts asteering angle of the vehicle, a brake device that generates a brakingforce for the vehicle, and the like.

The body system control unit 12020 controls operations of variousdevices which are mounted in a vehicle body in accordance with variousprograms. For example, the body system control unit 12020 functions as acontroller for a keyless entry system, a smart key system, a powerwindow device, and various lamps such as a head lamp, a back lamp, abrake lamp, a winker, or a fog lamp. In this case, radio waves emittedfrom a portable unit in place of a key or signals from various switchescan be input to the body system control unit 12020. The body systemcontrol unit 12020 receives inputting of such radio waves or signals andcontrols a door lock device, a power window device, lamps, and the likeof the vehicle.

The exterior information detecting unit 12030 detects information theexterior of the vehicle in which the vehicle control system 12000 ismounted. For example, an imaging unit 12031 is connected to the exteriorinformation detecting unit 12030. The exterior information detectingunit 12030 causes the imaging unit 12031 to capture an image outside thevehicle and receives the captured image. The exterior informationdetecting unit 12030 may perform an object detecting process or adistance detecting process for a person, a vehicle, an obstacle, a sign,characters, or the like on a road surface on the basis of the receivedimage.

The imaging unit 12031 is an optical sensor that receives light andoutputs an electrical signal corresponding to the light intensity of thereceived light. The imaging unit 12031 may output the electrical signalas an image or may output the electrical signal as distance measurementinformation. Further, light received by the imaging unit 12031 may bevisible light or may be invisible light such as infrared light.

The interior information detecting unit 12040 detects interiorinformation of the vehicle. For example, a driver state detecting unit12041 that detects a driver's state is connected to the interiorinformation detecting unit 12040. The driver state detecting unit 12041includes, for example, a camera that images a driver, and the interiorinformation detecting unit 12040 may calculate a degree of fatigue or adegree of concentration of the driver or may determine whether thedriver is drowsy on the basis of the detection information input fromthe driver state detecting unit 12041.

The microcomputer 12051 can calculate a control target value of thedriving force generating device, the steering mechanism, or the brakingdevice on the basis of the exterior or interior information of thevehicle acquired by the exterior information detecting unit 12030 or theinterior information detecting unit 12040 and can output a controlcommand to the drive system control unit 12010. For example, themicrocomputer 12051 can perform cooperative control for implementing afunction of an advanced driver assistance system (ADAS) includingcollision avoidance or shock mitigation of a vehicle, followingtraveling based on an inter-vehicle distance, constant-speed traveling,collision warning of a vehicle or lane departure warning of a vehicle,and the like.

Further, the microcomputer 12051 can perform cooperative control forautomatic driving or the like allowing a vehicle to travel autonomouslywithout depending on a driver's operation by controlling the drivingforce generating device, the steering mechanism, the braking device, orthe like on the basis of information around the vehicle which isacquired by the exterior information detecting unit 12030 or theinterior information detecting unit 12040.

Further, the microcomputer 12051 can output a control command to thebody system control unit 12020 on the basis of the exterior informationacquired by the exterior information detecting unit 12030. For example,the microcomputer 12051 can perform cooperative control for achieving anantiglare function by controlling the headlamp depending on a positionof a preceding vehicle or an oncoming vehicle detected by the exteriorinformation detecting unit 12030 and switching a high beam to a lowbeam.

The voice and image output unit 12052 transmits at least one outputsignal of a voice and an image to an output device that can visually orauditorily notify an occupant of a vehicle or the vehicle exterior ofinformation. In the example illustrated in FIG. 24, an audio speaker12061, a display unit 12062, and an instrument panel 12063 areillustrated as the output device. The display unit 12062 may include,for example, at least one of an onboard display or head-up display.

FIG. 25 is a diagram illustrating an example of an installation positionof the imaging unit 12031.

In FIG. 25, a vehicle 12100 includes imaging units 12101, 12102, 12103,12104, and 12105 as the imaging unit 12031.

The imaging units 12101, 12102, 12103, 12104, and 12105 are installed atpositions such as a front nose, a side-view mirror, a rear bumper, abackdoor, and an upper part of a front windshield of the interior of thevehicle 12100. The imaging unit 12101 provided in the front nose and theimaging unit 12105 provided in the upper part of the front windshield ofthe interior of the vehicle mainly acquire an image of the front of thevehicle 12100. The imaging units 12102 and 12103 provided in theside-view mirror mainly acquire an image of the sides of the vehicle12100. The imaging unit 12104 provided in the rear bumper or thebackdoor mainly acquires an image of the rear of the vehicle 12100. Theimages of the front acquired by the imaging units 12101 and 12105 aremainly used to detect a preceding vehicle, a pedestrian, an obstacle, atraffic sign, a traffic marking, a lane, or the like.

Incidentally, an example of imaging ranges of each of the imaging units12101 to 12104 is illustrated in FIG. 25. The imaging range 12111indicates imaging ranges of the imaging unit 12101 provided in the frontnose, the imaging ranges 12112 and 12113 indicates imaging ranges of theimaging units 12102 and 12103 provided in the side-view mirrors, and theimaging range 12114 indicates an imaging range of the imaging unit 12104provided in the rear bumper or the backdoor. For example, bysuperimposing image data captured by the imaging units 12101 to 12104,an overhead image showing the vehicle 12100 from the overhead isacquired.

At least one of the imaging units 12101 to 12104 may have a function ofacquiring distance information. For example, at least one of the imagingunits 12101 to 12104 may be a stereoscopic camera including a pluralityof imaging elements or may be an imaging element having pixels fordetecting a phase difference.

For example, the microcomputer 12051 can extract a three-dimensionalobject, which is the closest on the traveling path of the vehicle 12100and moves at a predetermined speed (for example, 0 km/h or more)substantially in the same direction as the vehicle 12100, as a precedingvehicle by calculating distances to three-dimensional objects within theimaging ranges 12111 to 12114 and change of the distances over time(speeds relative to the vehicle 12100) on the basis of the distanceinformation acquired from the imaging units 12101 to 12104. In addition,the microcomputer 12051 can set an inter-vehicle distance which shouldbe secured with respect to the preceding vehicle and perform automaticbrake control (including following stop control), automatic accelerationcontrol (including following start control), or the like. In this way,cooperative control for automatic driving or the like in which thevehicle travels autonomously without depending on a driver's operationcan be performed.

For example, the microcomputer 12051 can sort and extractthree-dimensional object data of three-dimensional objects intothree-dimensional objects such as a two-wheeled vehicle, a generalvehicle, a large vehicle, a pedestrian, a utility pole, or the like onthe basis of the distance information acquired from the imaging units12101 to 12104 and can use the data for automatic avoidance of anobstacle. For example, the microcomputer 12051 sorts obstacles near thevehicle 12100 into an obstacle which can be seen by a driver of thevehicle 12100 and an obstacle which cannot be seen by the driver. Then,the microcomputer 12051 can determine a collision risk indicating a risklevel of collision with each obstacle, and perform driving support forcollision avoidance by outputting a warning to the driver via the audiospeaker 12061 or the display unit 12062 or forcibly decelerating orperforming avoidance steering via the drive system control unit 12010when there is a likelihood of collision because the collision risk isequal to or higher than a set value.

At least one of the imaging units 12101 to 12104 may be an infraredcamera that detects infrared light. For example, the microcomputer 12051can recognize a pedestrian by determining whether or not a pedestrian ispresent in the captured images from the imaging units 12101 to 12104.Such recognition of a pedestrian can be performed, for example, by aroutine of extracting feature points in the captured images from theimaging units 12101 to 12104 as an infrared camera and a routine ofperforming a pattern matching process on a series of feature pointsindicating an outline of an object and determining whether or not theobject is a pedestrian. When the microcomputer 12051 determines that apedestrian is present in the captured images from the imaging units12101 to 12104 and recognizes a pedestrian, the voice and image outputunit 12052 controls the display unit 12062 such that a rectangular frameline for emphasis is displayed to overlap the recognized pedestrian.Further, the voice and image output unit 12052 may control the displayunit 12062 such that an icon or the like indicating a pedestrian isdisplayed at a desired position.

An example of a vehicle control system to which the technology accordingto the present disclosure can be applied has been described above. Thetechnology according to the present disclosure can be applied, forexample, the imaging unit 12031 in the above-mentioned configuration.Specifically, the imaging device 2 illustrated in FIG. 2 can be appliedto the imaging unit 12031. By applying the technology according to thepresent disclosure to the imaging unit 12031, the imaging unit 12031 canoutput information which is required by a user, that is, informationwhich is required by blocks performing processing in a subsequent stage(hereinafter also referred to as a subsequent-stage block). Accordingly,the subsequent-stage block does not need to perform a process ofgenerating necessary information from an image and it is thus possibleto reduce a load of the subsequent-stage block by as much.

Here, in this specification, the processes which are performed by acomputer (processor) in accordance with a program do not need to beperformed in time series in the order described as the flowcharts. Thatis, the processes which are performed by the computer in accordance witha program include processes which are performed in parallel orindividually (for example, parallel processes or processes usingobjects).

Further, the program may be processed by one computer (processor) or maybe distributed to and processed by a plurality of computers.

Incidentally, an embodiment of the present technology is not limited tothe above-mentioned embodiments and can be modified in various formswithout departing from the gist of the present technology.

For example, the present technology can be applied to an image sensorthat senses radio waves such as infrared light other than visible lightin addition to an image sensor that senses visible light.

Further, the effects described in this specification are only examplesand are not restrictive. Another effect may be achieved.

Incidentally, the present technology can have the followingconfigurations.

-   1. A single-chip imaging device including:

an imaging unit in which a plurality of pixels is arrangedtwo-dimensionally and that captures an image;

a signal processing unit that performs signal processing using acaptured image output from the imaging unit;

an output I/F that outputs a signal processing result of the signalprocessing and the captured image to an outside; and

an output control unit that performs output control of selectivelyoutputting the signal processing result of the signal processing and thecaptured image from the output I/F to the outside.

-   2. The imaging device according to <1>, in which the imaging device    has a stacked structure in which a plurality of dies is stacked.-   3. The imaging device according to <1> or <2>, further including an    image compressing unit that compresses the captured image and    generates a compressed image having a smaller amount of data than    that of the captured image.-   4. The imaging device according to any of <1> to <3>, further    including:

an imaging control unit that includes a register storing imaginginformation associated with capturing of the captured image and controlscapturing of the captured image according to the imaging information;and

an imaging information calculating unit that calculates the imaginginformation using the signal processing result,

in which the imaging control unit and the imaging informationcalculating unit are connected to each other via a predeterminedconnection line, and

the imaging information calculating unit feeds back the imaginginformation to the register of the imaging control unit via thepredetermined connection line.

-   5. The imaging device according to <4>, in which the register stores    output control information associated with the output control, and

the output control unit performs the output control according to theoutput control information stored in the register.

-   6. The imaging device according to <4> or <5>, further including a    first communication I/F that exchanges information to be read from    and written to the register with the outside.-   7. The imaging device according to any of <1> to <6>, in which the    signal processing unit is a processor that executes a program, and

the imaging device further includes a second communication I/F thatdownloads the program which is executed by the processor from theoutside.

-   8. The imaging device according to any of <1> to <7>, in which the    signal processing unit performs a recognition process of recognizing    a predetermined recognition object from the captured image as the    signal processing.    9. The imaging device according to any of <1> to <7>, further    including an input I/F that receives an external sensor output from    an external sensor,

in which the signal processing unit performs signal processing using thecaptured image and the external sensor output.

-   10. The imaging device according to <9>, in which the external    sensor output is an output of a range sensor that senses information    associated with a distance or an output of an image sensor that    senses light and outputs an image corresponding to the light.-   11. The imaging device according to <10>, in which the signal    processing unit performs a fusion process of calculating a distance    using the captured image and the output of the range sensor or a    self-localization process using an image as the output of the image    sensor and the captured image as the signal processing.-   12. An electronic device including:

an optical system that collects light; and

a single-chip imaging device that receives light and outputs an imagecorresponding to a received light intensity of the light,

in which the imaging device includes

an imaging unit in which a plurality of pixels is arrangedtwo-dimensionally and that captures an image,

a signal processing unit that performs signal processing using acaptured image output from the imaging unit,

an output I/F that outputs a signal processing result of the signalprocessing and the captured image to an outside, and

an output control unit that performs output control of selectivelyoutputting the signal processing result of the signal processing and thecaptured image from the output I/F to the outside.

REFERENCE SIGNS LIST

-   1 Optical system-   2 Imaging device-   3 Memory-   4 Signal processing unit-   5 Output unit-   6 Control unit-   20 Imaging block-   21 Imaging unit-   22 Imaging processing unit-   23 Output control unit-   24 Output I/F-   25 Imaging control unit-   26 Communication I/F-   27 Register group-   30 Signal processing block-   31 CPU-   32 DSP-   33 Memory-   34 Communication I/F-   35 Image compressing unit-   36 Input I/F-   51, 52 Die-   71 Rectification unit

1. An imaging device, comprising: an imaging unit in which a pluralityof pixels is arranged two-dimensionally and that captures an image; arecognition processor that performs at least part of a recognitionprocess for a first captured image output from the imaging unit; anoutput interface (I/F) that outputs both a recognition result by therecognition processor or intermediate data acquired midway of therecognition process and a second captured image to an outside of theimaging device; and an output control processor that causes at least oneof the recognition result or the intermediate data and the secondcaptured image to be selectively outputted from the output 1/F to theoutside of the image device, wherein the imaging unit and therecognition processor are arranged on a single chip.
 2. The imagingdevice according to claim 1, wherein the imaging device has a stackedstructure in which a plurality of dies is stacked.
 3. The imaging deviceaccording to claim 1, further comprising an image compressing unit thatcompresses the first captured image and generates a compressed imagehaving a smaller amount of data than that of the first captured image.4. The imaging device according to claim 1, further comprising: animaging control processor that includes a register storing imaginginformation associated with capturing of the first captured image andcontrols capturing of the first captured image according to the imaginginformation; and an imaging information calculating processor thatcalculates the imaging information using the recognition result, whereinthe imaging control processor and the imaging information calculatingprocessor are connected to each other via a predetermined connectionline, and the imaging information calculating processor feeds back theimaging information to the register of the imaging control processor viathe predetermined connection line.
 5. The imaging device according toclaim 4, further comprising a first communication I/F that exchangesinformation to be read from and written to the register with the outsideof the imaging device.
 6. The imaging device according to claim 1,wherein the recognition processor executes a program, and the imagingdevice further comprises a second communication I/F that downloads theprogram which is executed by the recognition processor from the outsideof the imaging device.
 7. The imaging device according to claim 1,further comprising an input I/F that receives an external sensor outputfrom an external sensor, wherein the recognition processor performsrecognition processing using the first captured image and the externalsensor output.
 8. The imaging device according to claim 7, wherein theexternal sensor output is an output of a range sensor that sensesinformation associated with a distance or an output of an image sensorthat senses light and outputs an image corresponding to the light. 9.The imaging device according to claim 8, wherein the recognitionprocessor performs a fusion process of calculating a distance using thefirst captured image and the output of the range sensor or aself-localization process using an image as the output of the imagesensor and the first captured image as the recognition processing. 10.An electronic device comprising: an optical system that collects light;and a single-chip imaging device that receives light and outputs animage corresponding to a received light intensity of the light, whereinthe imaging device includes: an imaging unit in which a plurality ofpixels is arranged two-dimensionally and that captures an image; arecognition processor that performs at least part of a recognitionprocess for a first captured image output from the imaging unit; anoutput interface (I/F) that outputs both a recognition result by therecognition processor or intermediate data acquired midway of therecognition process and a second captured image to an outside of theimaging device; and an output control processor that causes at least oneof the recognition result or the intermediate data and the secondcaptured image to be selectively outputted from the output I/F to theoutside of the image device, wherein the imaging unit and therecognition processor are arranged on a single chip.
 11. The imagingdevice according to claim 1, wherein the single chip includes a firstsubstrate and a second substrate joined together, wherein the firstsubstrate includes the imaging unit, and wherein the second substrateincludes the recognition processor.
 12. The imaging device according toclaim 1, wherein the first captured image is an image generated bycompressing the output of the imaging unit, and wherein the recognitionprocessor performs at least part of the recognition process for thefirst captured image generated by the compression.
 13. The imagingdevice according to claim 1, wherein the first captured image is animage generated by performing a gradation process for the output of theimaging unit.
 14. The imaging device according to claim 1, furthercomprising: an imaging processor that processes the first captured imageoutputted from the imaging unit; and a control processor that controlsprocessing by the imaging processor or an imaging operation to beexecuted by the imaging unit, according to the result of the recognitionprocess.
 15. The imaging device according to claim 14, wherein thecontrol processor causes the imaging processor to execute a process ofcutting out a part associated with an area specified by the recognitionprocess from the first captured image.
 16. The imaging device accordingto claim 14, wherein the control processor controls the imagingoperation by the imaging unit such that a part associated with an areaspecified by the recognition process is read out.
 17. The imaging deviceaccording to claim 16, wherein the control processor controls at leastone of an exposure time, a focus, a frame rate, and an imaging mode inthe imaging operation by the imaging unit.
 18. The imaging deviceaccording to claim 1, wherein the recognition processor performs therecognition process using a technique of deep learning.
 19. The imagingdevice according to claim 18, wherein the technique of deep learning isa convolutional neural network.
 20. The imaging device according toclaim 1, wherein the recognition processor executes the recognitionprocess after acquisition of the first captured image based on theoutput of the imaging unit being completed in each frame.